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ABSTRACT 

The prima ry p urpose of the present study was to 
investigate empirically the effect of multiple hypothe-sis testing on 
error rates in factorial ANOVA designs under a variety of controlled 
conditions. The per comparison, per experiment, and experimentalise 
error rates were investigated for three hypothesis testing 
procedures. The specific conditions manipulated included: (1) the 
number of factors in "be design, (2) the number of levels of each 
factor, (3) the number of observations per cell, and (4) the 
population values of each null hypothesis (magnitude of the effect) • 
A Monte Carlo computer simulation procedure was used for generation 
of the data. Type I and Type II errors were tabulated .where 
appropriate fof the three error rates. (Author) 
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Error Rates of Multiple F Tests 
In Factorial ANOVA Designs 

The concept of error rates in hypothesis testing was 
introduced to help the experimenter estimate the frequency 
of erroneous inferences. With one statistical test, error 
rates can be estimated if one knows the significance level 
and direction of the test as well as the sample size used. 
With more than one statistical test, two other factors 
must be considered^ the number of hypotheses tested and the 
dependence among the tests. 

The number of hypotheses tested in an experiment is 
extremely important in multiple hypothesis testing because 
of the accumulation of errors. For example, with 100 inde- 
pendent tests of true null hypotheses, each tested at the 
.05 level of significance, it would be expected that 5 of 
the tests would be significant just by chance alone. Error 
rates are also affected by the dependence among the tests, 
such as in tests of all pair-wise comparisons based on a 
set of independent means. 

The problems of multiple hypothesis testing and their 
effects on error rates have been thoroughly investigated 
for the case of several independent means such as in the 
context of one-way analysis of variance (ANOVA) designs 
(Tukey, 1953; Ryan, 1959; Games, 1971). The same problems 
of multiple hypothesis testing also exist with higher-order 
ANOVA designs. 

In complex ANOVA designs, many hypotheses may be 
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may be tested within a single experLr.--:.' : .r example, 
in a five-factor fixed-effects design, t:.-.;i e ::;ay be as many 
as 31 hypotheses. With each hypothesis tested at the .05 
level of significance, and assuming all null hypotheses 
are true, it would be expected that an experimenter would 
be making l # 5 Type I errors per experiment. Higher-order 
ANOVA designs also have the problem of dependence among 
the tests. Although it is true that the numerator sums 
of squares are based on independent components of the total 
sum of squares (Lindquist, 1953), the tests themselves may 
be based on the same mean square for error; thus, chere 
may he dependence among the tests which could affect the 
frequency of errors within the total design (Hays, 1963). 

At the present time, very little is known about the 
effects of multiple hypothesis testing on error rates in 
multi-dimensicnal ANOVA designs. Empirical research has 
centered on the problem of comparisons based on a set of 
independent means as in one-way designs (Petrinovich & 
Hardyck, 1969; Norton & Bulgren, 1965) • Theoretical dis- 
cussions of an appropriate error rate to describe the fre- 
quency of errors in multiple testing situations have also 
centered on the one-way context (Tukey, 1953; Ryan, 1959; 
Games, 1971) . 

Purpose 

The purpose of this study was to empirically investi- 
gate the frequency of erroneous conclusions in factorial 
ANOVA designs under a variety of controlled conditions. 

ERLC 
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The frequency of errors was measured using the following 
three error rates: the error rate per test, the error rate 
per experiment, and the experiment wise error rate (Tukey, 
1953; Hartley, 1955; Ryan, 1959). Since the method of 
testing each hypothesis affects the frequency of erroneous 
conclusions, three different hypothesis testing procedures 
were used: (a) test each hypothesis at a specified alpha 
( c< ) level of significance, (h) use Hartley's (1955) 
sequential testing procedure"'" designed to control the 
experiment wise error rate at a specified ot> level, and 
(c) test each hypothesis at the ck /k level for k tests 
according to a Bonferroni procedure (Miller, 1966; Games, 
1971) • Factorial designs were varied according to the 
number of factors, the number of levels of each factor, 
the number of observations per cell, and the population 
values of the null hypotheses (all true, all false, and 
combinations of both true and false in the same design) . 
Where appropriate, Type I and/or Type II error rates were 
calculated. In all cases, was s»jt at .05. 

Procedure 

Both two- and three-way completely crossed fixed 

effects factorial designs with independent groups per cell 

were studied. The designs selected were the 2x2, 2x3? 2x4, 

2x5> 5x5, 2x2x2, and 5x5x5 designs. For all designs, the 

number of observations per cell were equal. 

The data were randomly generated by computer using 

2 

a Monte Carlo simulation procedure. All data were drawn 
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from normal distributions with equal variances and were 
generated according to the general linear model for ANOVA 
designs. The values of the main and interaction effects 
were calculated using Cohen's (1969) f index of effect size* 
The magnitudes of ef feces were varied across four points; 

zero (f = .00), small (f = .10), medium (f = .25), and large 
(f s ,40). For all designs, the main and interaction effect 
sie.es were held constant. In addition, for the 2x4 and 
2x2x2 designs effect size was varied across main and inter- 
action effects. 

A single simulation procedure consisted of the gener- 
ation of one set of scores for a single design under com- 
binations of the following conditions: (a) dimensions of 
the design, (b) cell size, and (c) population value of each 
effect. Acceptance or rejection of each hypothesis was 
determined using three criteria; (a) the usual F procedure 
with <£. = .05 as the significance level for each test, 
(b) Hartley's sequential procedure, and (c) a Bonferroni 
procedure with ,0 5/k as the significance level for each 
test. Each simulation procedure was repeated 2000 times. 
Following the 2000 replications, the following three error 
rates were calculated for each of the hypothesis testing 
procedures: (a) per comparison error rate or the average 
of the individual hypothesis error rates, (b) per experiment 
error rate or the average number of errors per experiment, 
and (c) the experiment wise error rate or the proportion of 
experiments with at least one error. 
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Results 

Per comparison error rates . Table 1 presents the per 
comparison error rates for the true null condition (all 
f f s = .00 in a given design) • The per comparison error 
rates of the alpha procedure fluctuated around the nominal 
•0$ level with a median value of .0 520. When <^ /k was 
used to test each hypothesis, the per comparison error rates 
were close to the expected values: the median per comparison 
error rate for two-way designs was .0172 (nominal level of 
.05/3 = .0167); the median was .0071 for three-way designs 
(nominal level of .05/7 = .0071). The per comparison error 
rates obtained by Hartley's procedure were just slightly 
higher than those obtained using the Bonferroni procedure. 
For two-way designs, the median error rate using Ha tley's 
procedure -was .017S; with three-way designs, the median 
was .0072. 

The per comparison error rates for all null hypotheses 
false (alJ f f s > .00) are given in Table 2. Under these 
conditionr, the per comparison error rate is the average of 
the Type II error rates of the individual hypotheses. One 
minus the per comparison error rate is the average power of 
the tests. 

With all three hypothesis testing procedures, the Type 
II error rates were affected by the total sample size and 
the size of the effects in the design. As the sample size 
increased, either by increasing the cell size or by increas- 
ing the number of levels or factors with a constant cell 

O 
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size, the per comparison error rates declined. As the mag- 
nitude of the effects increased, the number of errors decreased* 
That is, all three procedures were the most powerful with 
large sample sizes and large effects present. With small 
sample sizes and small effects, all three procedures had 
almost equal difficulty detecting small deviations from the 
true null condition* In the most extreme case of large sample 
sizes and large effects (5x5 design with n = 30 and all 
effects defined by f = .40), all procedures were equally 
capable of rejecting the null hypotheses 

In general, the fewest Type II errors were made using 
the alpha procedure; the most were made using c< /k. The 
frequency of errors made using Hartley's procedure was 
closer to that obtained using <=< /k when effect sizes were 
small but approached the Type II error rate of the alpha 
procedure as effect size and sample sizes increased. 

When both true and false null hypothese were present 
in the same design, both Type I and Type II per comparison 
error rates were calculated. The results are presented 
in Table 3 for the 2x4 designs and Table 4 for the 2x2x2 
designs. The per comparison error rates were calculated 
as averages across all hypotheses. The frequency of Type 
I errors closely followed the previous results: the alpha 
procedure returned the most Type I errors and the ck /k 
procedure returned the fewest . Because of the sequential 
nature of Hartley's procedure, the status of all hypotheses 
affected the average error rates. Thus, as the proportion 

ERLC 
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of false null hypotheses increased, the Type I error rate 
of Hartley 1 s procedure approached the level obtained by the 
alpha procedure. 

Per experiment error rates . The per experiment error 
rate is the average number of errors per experiment. It is 
also equal to the number of hypotheses times the per compari 
son error rate. Because of this direct relationship with 
the per comparison error rates, the per experiment error 

3 

rates will not be discussed in detail nor reproduced here. 

For all true null hypotheses, the average number of 
errors per experiment was close to k times the nominal level 
of significance used to test each hypothesis. For the two- 
way designs, the per experiment rates for the <^ = »05 
procedure were close to .15. Using .05/3, the error rates 
were close to .0 5. For three-way designs, the average 
number of errors using ck = .05 was close to .35- The 
Bonferroni procedure maintained a per experiment rate of 
close to .05* For all designs, Hartley's procedure produced 
just slightly more Type I errors than the Bonferroni pro- 
cedure. 

The results obtained when all effects were non-zero 
and when the effects varied in a design paralleled those 
presented on per comparison error rates. 

Experiment wise error rates . The experiment wise error 
rates when all effects were set at zero are presented in 
Table 5- These error rates are equal to the proportion of 
experiments with at least one Type I error. 
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When all null hypothese were true, the experimentwise 
error rates using Hartley 1 s procedure and the Bonferroni 
procedure were identical. That is, if one hypothesis were 
rejected using the c< /k level in the Bonferroni procedure, 
then that same hypothesis was also rejected by Hartley's 
procedure, which begins its sequential testing at the c< /k 
level. Both of these procedures controlled the experiment- 
wise error rates at the .05 level for both two- and three- 
way designs. When <A = .0$ was used to test each hypothesis, 
the frequency of errors increased as the number of tests 
increased. With three hypotheses in the two-way designs, 
the experimentwise error rates had a median value of .1460. 
With seven tests in three-way designs, the median experi- 
mentwise error rate of the alpha procedure jumped to .2970. 

With all null hypotheses false, the experimentwise errcr 
rate was equal to the proportion of experiments v/ith at least 
one Type II error (see Table 6). With Type II errors, all 
testing procedures had different experimentwise error rates. 
The Type II error rates followed the same trends as the Type 
II per comparison error rates. In all cases, the alpha 
procedure had the lowest experimentwise error rate and 
the Bonferroni procedure had the highest rate. The power 
of all three procedures increased as the total number of 
observations and the sizes of the effects increased. 

With both true and false null hypotheses in the same 
design, two experimentwise error rates were calculated: the 
proportion of experiments with at least one Type I error 



and the proportion with at least one Type II error. These 
error rates are presented in Tables 7 and £>. All error rates 
were affected by the number of false null hypotheses and 
followed trends already noted for Type I and II experimentwise 
error rates • As with the per comparison error rates, the 
experimentwise error rates of Hartley's procedure were compli- 
cated by the sequential nature of the test. 

Discussion 

This study has empirically investigated the frequpncy 
of Type I and Type II errors in selected factorial ANOVA 
designs* The frequency of errors was measured by three 
different error rates: the per comparison error rates or 
the average of the individual hyyjthesis rates and two 
experiment-based error rates, the per experiment error 
rates and the experimentwise error rates. 

V/hen the significance level is sat at #05 for each 

hypothesis, a practice commonly followed in educational 

research, the accumulation of Type I errors as measured by 

the per experiment and experimentwise error rates was 

readily apparent. In this study, the per experiment error 

rates for all designs were close to k times .05 for k tests* 

The experimentwise error rates were close to .1$ and .30 

for two- and three-way designs, respectively. It should 

be noted that these experimentwise error rates are close 

to the expected experimentwise error rates of k independent 

k 

tests as estimated by the formula 1 - (1 - eO— where 

is the level of significance used for each hypothesis. 
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Thus, it appears that the dependence among the tests in 
factorial designs when the same mean square for error is 
used with each test has little, if any, effect on the 
experimentwise error rates. In addition, the estimate 
based on k independent tests appears to be valid with 
small as well as large degrees of freedom for error. The 
alpha procedure, then, controls the individual hypothesis 
error rates but allows the experiment -based error rates to 
increase as the number of tests increases. 

The Bonferroni and Hartley procedures adjusted the 
individual hypothesis significance levels to maintain the 
experiment-based error rates at acceptable and pre-specified 
levels. The Bonferroni procedure used /k for the sig- 
nificance level of each hypothesis while Hartley's procedure 
used different significance levels for each hypothesis 
based on their obtained £ values. The individual hypothesis 
rates of both procedures were close to <^ /k, thus adjusting 
the individual hypothesis rates as k increased. The per 
experiment and experimentwise error rates of these two pro- 
cedures were held at the .05 level in both two- and three- 
way designs. 

The frequency of Type II errors depended on the total 
number of observations and the sizes of the effects present. 
All three procedures had difficulty detecting small effects 
with small sample sizes; all were capable of detecting large 
effects with large sample sizes. 
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The choice of an appropriate hypothesis testing tech- 
nique depends on the importance of experiment-based errors. 
If individual hypotheses are of major concern rather than 
a pattern of results obtained from an entire design, then 
a procedure which controls the individual hypothesis error 
rates at acceptable levels, such as the alpha procedure 
used in the present study, would be appropriate. However, 
if the accumulation of possible errors across an entire 
design would severely limit the validity of an experiment, 
then a procedure that controls the experiment -based error 
rates, such as a Bonferroni procedure or Hartley's pro- 
cedure for factorial designs, would be more appropriate. 
The choice between individual hypothesis error rates and 
c v pc^*imc t b c c c d cr^c^ rates has been debated elsewh^r^ 
(Miller, 1966; Ryan, 1962; Wilson, 1962; Petrinovich & 
Hardyck, 1969) with no apparent resolution. Regardless 
of which type of error an experimenter chooses to control, 
he should be aware of the accumulation of errors that 
result from multiple hypothesis testing. 
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Footnotes 

1. Hartley's (1955) procedure tests each hypothesis in 
a factorial design based on its obtained £ value. All 
hypotheses are ranked according to their p values and the 
hypothesis with the smallest p value is tested for signi- 
ficance at the °Vk level where k is the number of hypotheses 
in the design. If the hypothesis is significant, the next 
ranking hypothesis is tested at the ^/(k-l) level. The 
testing continues until a hypothesis is not rejected. At 
that point, testing stops and all remaining hypotheses are 
declared non-significant. Hartley demonstrates how this 
sequential procedure maintains an experiment wise error 
rate at c*. . 

2. A more detailed description of the computer simulation 
procedure t including information concerning the random numbe 
generator used, can be found in Halderson, 1973. 

3. Tables of per experiment rate? can be found in Halderson 
1973. 



Table 1 
Per Comparison Krror Rates 
All f's = 0 



Design 


n 


Procedure 


Alpha a 


3onf erronir 


Hartley 0 


j"w <J 


15 
30 


.0572 
.0583 


.0183 
.0180 

01 87 


.0195 
.0188 
-.Dion 




15 
30 


.0455 
.0542 

a f o Q 
• 052O 


.0152 

.0187 
.017o 


.0153 

.0193 
• 0178 


^X*+ 


5 
15 
30 


.0543 
.0520 

.0523 


.0190 

.0155 
• 01 o<c 


.0200 
.0157 
• ui 03 




< 

15 
30 


.0502 
.0490 
.0502 


.0180 
.0155 
.0155 


.0188 
. 0162 
.0158 


5x5 


5 
15 
30 


.0457 
.0533 
.0547 


.0145 
.0172 
.0172 


.0150 
.0180 

.0177 


2x2x2 


5 

15 
30 


.0515 
.0488 

.0530 


.0077 
.0054 
.0071 


.0079 
.0054 
.0071 


5x5x5 


5 


.0514 


.0071 


.0072 



a Nominal level of .05 for each hypothesis. 

^Nominal level of .05/k for each hypothesis; k is the 
number of tests in any one experiment. 

c Nominal level for each hypothesis is not known. 



Table 2 
Per Comparison Error Rates 
All f's > 0 



Des ign 


f 


n 


Error Rates 




a 

( Alpha 


Bonf erroni b 


Hartley 0 


2x2 


.10 


5 
15 
3 U 


• 934? 
.3863 
.8265 


.9747 
• 9557 
.9160 


.9738 

• 9537 

• 9125 




.25 


5 
15 
30 


.3333 
• 5783 
.306O 


• 9275 
.7395 
.4635 


.921 2 

• 7067 
.3818 




.40 


5 
15 
30 


• 6575 
.2255 
.0492 


.8142 
.3672 
.0952 


.7880 

.2785 
.0503 


2x3 


.10 


5 
15 
30 


.9245 
.8782 

.7975 


.9712 

• 9^55 
.8992 


.9693 
.9430 
.8923 




.25 


5 
15 
30 


.0065 

.4912 
.2155 


.9073 
.6557 
.3425 


.8988 
.6113 
• 2553 




.40 


5 
15 
30 


.5910 
.1505 

.0233 


.7502 
.2498 
.0505 


.7210 
.1702 
.0248 


2x4 


.10 

| 


5 
15 
30 


.9272 
.8700 
.7608 


.9742 
.9438 
.8692 


.9723 
• 9417 
.8600 




• 25 


5 
15 
30 


.7755 

.4215 
.1712 


.3395 
.5795 
.2670 


.3767 
• 5185 
.1945 




.40 


5 
15 
30 


.5120 

.1075 
.0098 


.6753 
.1355 
.0263 


.6282 
.1147 
.OO98 



Nominal level of ,05 for each hypothesis. 

^Nominal level of .05/k for each hypothesis; k is the 
number of tests in any one experiment. 

°Nominal level for each hypothesis is not known* 



Table 2 
(Continued) 



Design 


f 


n 


Error Kates 


Alpha 


3onrerroni 


Hartley 


2x5 


.10 


5 
15 
30 


.9200 

• 7313 


• 9717 
.8562 


.9700 

no/in 

.8435 




.25 


5 
15 
30 


.7402 

• 3/30 
.1302 


.8632 
.2112 


.8503 
.1415 




.40 


5 
15 
30 


.^537 
• O693 
.0055 


.6068 

• 1 31° 

.0145 


• 5535 

• ^ /l / 
.0055 


5x5 


.10 


5 
55 
30 


.8937 
• 7533 
.5507 


• 9578 
. 8618 
.6930 


•9565 
.8505 

•6573 




.25 


5 
15 
30 


.5435 
.1207 
.OO87 


.7002 
.1930 
.0233 


.6615 
• 129? 
.0087 




.40 


5 
15 
30 


.1823 

.0000 


.2778 

• u\J (5 
.0000 


.2050 

A AO O 

.0000 


2x2x2 


.10 


5 
15 
30 


.9154 

. 8491 
.7443 


.935^ 
.9634 

.9184 


.9852 

.9623 

.9136 




.25 


5 
15 
30 


.7458 
.4071 
.1723 


.9246 
.6840 
• 3746 


.9198 

.6286 

.2601 




.40 


5 
15 
30 


.4893 
.1143 
.0236 


.7569 
.2767 
.0731 


.7209 
.1618 
.0251 
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Table 3 
Per Comparison Error Rates i 
2r.k Designs with Various f *s 











Typo I 


Type II 








n 


A 1 phct 1 


3on.t u . 


Hart ley 0 




3onf P 


Hartley 


.25 


0 


0 


5 
15 

30 


. 03U0 
.0310 
.0323 


.011 5 
. 0092 
.0128 


.0130 
• 0133 
.0173 


.220? 
.0763 
.0097 


.2710 
.1272 
.0233 


.2707 
.1272 
.0282 


0 


.25 


0 


5 
15 

30 


.0353 
.0320 
.0355 


.0117 
.0102 
.0117 


.0130 
.0120 
.0169 


.2620 
.1315 
.0267 


.3002 

.1955 
.0567 


.2993 
.1950 
.0560 


.25 


0 


.25 


5 
1 5 
30 


.0l?0 
.017S 
.0160 


.0057 
.0073 
. 00*47 


.0070 
.0105 
.0122 


.5037 
.2962 

.1*4-32 


• 5823 
.3930 
.2073 


•5793 
• 3795 
.1383 




0 


.25 


5 
15 
30 


.0172 
.0167 
.0187 


. 0052 

.0065 
.00*18 


. 0088 
. 0093 
.0147 


. 3S62 
.2275 
.1303 


.*+7l2 
.2810 
.18*47 


. *4-63? 
.2665 
.1653 


0 


.25 


.25 


5 
15 
30 


.0165 
.01.37 
.0192 


.0057 
.0052 

.0067 


.0062 
.0067 
.0130 


.5518 
•3557 
• 1595 


.6168 
.*4630 
.2*452 


.61*43 

.*4*49S 
.2225 


0 


.k-0 


• 25 


5 
15 
30 


.0160 
.0102 
.0180 


.0052 
.0065 
.OO63 


.0072 
.0112 

.01^-3 


.^562 
.2320 
.1265 


.5*462 
.2903 
.1865 


.5**20 

.3733 
.16*43 



a Nomiraju level of .05 for each hypothesis. 

^Nomii.al level of . 05/k for each hypothesis} k is the 
number of tests in any one experiment- 

c Nominal level for each hypothesis is not known. 
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Table 4 

Per Comparison Error Rates « 
2x2x2 Designs with Various f's 











Type I 


Type II 




S : 'A 






A 1 p h ; 9 


Bonf ? 


i".rirtley u 


Alpha 0 




.iartley 


.25 


0 


0 


5 

15 
30 


. 0465 
.0475 


An <T 
• UU ) J 

.0060 

. 0054 


• U U j L r 

.0063 
.0062 


.0316 
.0649 


• 1 £ 2 r 

.071 9 

.0181 


.0717 
.0181 


.25 


.25 


0 


5 

1 5 
30 


.036 r 
.04l4 
.0404 


. 004o 
.0054 
. 0056 


. 0 0 c , 1 

• v V' J I 

. 0063 
. 0076 


. 21 06 

.1042 
.0351 


.2626 
. 1825 
.0899 


.2622 

• 1799 
. 08 54 


.40 


.25 


0 


5 
15 

O Pi 


.0339 
.0 364 
.0381 


. 0054 
. 0056 
.0049 


. 00 57 
. 0074 
.0074 


. 1 551 
.0739 
.0292 


.2202 
• 1193 
.0699 


. 2194 
.1171 
.0656 


.25 


.25 


.25 


5 
15 
30 


.0301 
.0272 

,0 344 


.0032 

.0039 
.0040 


.0034 

.0046 
.0050 


•3321 
.2078 
.1051 


.4001 

• 3119 

• 1979 


• 3991 
. 3078 
. 1 889 


.40 


' .40 


■ 25 


c 

15 

30 

L_ . 


.0279 
.0310 


.0039 

.00^5 


n r> £ 2 
.OO56 
.0073 


!l24S 
.074i 


>>ho« 

• 

.1899 
.1190 


• 3459 
.1823 

.1129 



d 



a All additional effects set at zero; that is, 



f "p *— f r\ f • n *~* f t-> — 0 • 

b Nor,inal level of .05 for each hypothesis. 

°Non,inal level of •05/k for each hypothesis; k is the 
number of tests in any one experiment. 

^Nominal level for each hypothesis is not known. 



Table 5 
Experimentv/ise Error Rates 
All f»s = 0 



Des ign 


n 


Proc edure 


Alpha 3- 


Bonf erro.nl/ 
Hartley 0 


2x2 


5 

30 


.1550 
.1640 
.1425 


.0535 
.0535 
.0550 


2x3 


5 
15 

30 


.1275 
.1545 
.1525 


.0440 
.0545 
.0535 


2x4 


5 

1 D 

30 


.1480 
.1470 
. 1490 


.0555 
.0455 
.0480 


2x5 


5 
15 

•an 


.1445 
.1385 

. 142C 


.0530 
.0460 


5x5 


5 
15 
30 


.1255 
.1515 
.1545 


.0435 
.0515 
.0510 


2x2x2 


5 
15 
30 


.2920 

.2955 
.3160 


.0500 
.0370 
.0490 


5x5x5 


5 


.2935 


.0485 



a Nominal level of .05 for each hypothesis; experimentwise 
rate not known. 



D ExperiTnentv/ise error rates are identical and equal to 
• 05 using these two methods when all f *s = 0. Bonferroni 
uses a nominal level of -05/^ for each hypothesis. Nominal 
level for each hypothesis for Hartley is unknown. 



Tabic 6 
Experimentwise Error Rates 
All f «s > 0 



Design 


f 


n 




Error Rates 




Alpha a 


3onf crroni 


HartlPjF 


2x2 


.10 


5 
15 
30 


• 9995 
•9955 
.9960 


.9995 
.9990 

• 9995 


.9995 
.9935 
.9980 




.25 


5 
15 
30 


.9905 
.9280 

.6885 


.9985 
.9795 
.8680 


.9925 
.9400 

.7075 




.40 


5 
15 
30 


.9475 
.5480 

.1465 


.9845 
.7590 
.2755 


• 9555 
.5675 
.1475 


2x3 


.10 


5 
15 
30 


• 9975 

• 9975 
.9965 


.9995 
• 9995 
1 .0000 


.9930 
.9975 
.9970 




.25 


5 
15 
30 


• yo'o 

.8815 
.5485 


.9980 
-9600 
.7660 


.9900 
.8985 
-5655 




.40 


5 
15 
30 


.9250 
.4110 
.0715 


.9745 
.6125 

.1485 


.9365 
.4190 

.0725 


2x4 


.10 


5 
15 
30 


.9990 

• 9975 
.9910 


1 .0000 

• 9995 
.9990 


• 9995 
.9990 
.9940 




.25 


5 
15 
30 


.9345 
.8285 

.4735 


• 9975 

• 9425 
.6660 


.9380 
. 8440 
.4835 




.40 


5 
15 
30 


.8885 
.3080 
.0295 


.9610 
.5010 
.0790 


.8975 
• 3115 
.0295 



a Nominal level of .05 for each hypothesis* 



Nominal level cf • 05/i£ for each hypothesis 1 k is the 
number of tests in any one experiment. ~ 

°Nominal level for each hypothesis is not known. 



Tab] e 6 
(Continued ) 



Design 


f 


n 


Error Rates 


Alpha 


3onf erroni 


Hartley 


2x5 


.10 


5 
15 
30 


• 9975 
.9360 


1 .0000 
1 . 0000 

J. i w \y u v 

• 9985 


.9985 
. QQQ0 

• 9885 




.25 


5 
15 
30 


• 9795 

• 787 

. 3720 


.9985 
. 91 7 *) 

.5645 


.9855 
.8015 

.3770 




.40 


5 
15 
30 


.8450 
.0165 


.9395 

77^ 

.0435 


.8575 

• ij \J _J \J 

.0165 


5x5 


.10 


5 
15 
30 


.9970 
.9225 


.9995 

.9305 


.9990 
.0880 

•9335 




.25 


5 
30 


• 9165 
. 3500 

.0260 


.9800 

. ^260 
.6706 


.9305 

_ T C)\.C 

.0260 




.40 


5 
15 
30 


.5040 
. OO65 
.0000 


.6985 
.0225 
.0000 


.5140 
.0065 
.0000 


2x2x2 


.10 


5 
15 
30 


1.0000 

1.0000 
1.0000 1 


1 .0000 
1 .0000 
1 .0000 


1.0000 

1 .0000 
1 . 0000 




.25 


5 

15 
30 


1.0000 

.9730 
.7750 


1 .0000 
1.0000 

.9755 


1 .0000 

.9365 
.3060 




.40 


5 
15 
30 


.9905 
.6060 
.1605 


1 .0000 

.9295 
.4530 


.9950 
.6330 
.1625 



9 

ERIC 



Table 7 
Expcrino.ntv/ise Error Rates t 
2x4 Designs with Various f f s 



—A 


f. 3 

ra 


f . p 


!1 


Typo I 


Type 11 


Alpha a 




Hartley 0 


Alpha' 1 




hartley 1 - 


.25 

0 


0 


0 


5 

15 

on 
J U 


. 09?0 
. 0005 
.0950 


.0340 

.0275 
.0380 


.0375 
.03SO 
.0500 


.6620 
.22^0 
.0290 


.8130 

.3815 

. 0700 


.81 20 
• 3 q l5 
.0695 


.25 


0 


5 
15 
30 


.0995 
.0925 

.1050 


,0335 
.02Q^ 
.0350 


.0370 
.0340 
.0500 


.7860 

• 3945 
.0300 


.9005 
.5865 

.1700 


.8980 
• 5850 
.1600 


.25 


0 


.25 


5 
15 

30 


.0510 

• 0535 
.0430 


.0170 
.0220 
.O35O 


. 021 0 
.0315 
.0365 


.9470 
.7400 
.4185 


.9835 
.8755 
.5895 


.9760 
• 8375 
.5325 


.40 


0 


.25 

.25 


5 
\5 
30 

5 

30 


• u J J .} 

.0500 

.0560 


. 01 q 5 
.0145 


()?(•) ^ 

m \J tj <.) _j 

. 0295 
. 0440 


801 0 

-6775 
.3910 


.9680 
.8215 

.5540 


• j r jj 
.7780 
.4960 


0 


.25 


.0495 
.0575 


.0170 

n 1 r r 
.0200 


.01 85 

.0390 


.9625 

O ^ f f 

• W w W J 

.4380 


.9915 

• 9 2 3 j 
.63OO 


.985c 

• ■-■'■'■J. 
.5625 


0 


.40 


.25 


5 

15 
30 


.04% 

.0575 
.0540 


.0155 

.0195 
.0190 


.0215 

.0335 
.0430 


.9260 
.6715 
.3795 


.9780 
. 3065 
•5595 


.9675 

.7575 
.4930 



a Nominal level of .05 for each hypothesis. 

^Nominal level of . 05/k for each hypothesis; k is the 
number of tests in any one experiment. 

°Nominal level for each hypothesis is not known. 



Table 8 

Experimentv/ise Error Rates i 
2x2x2 Designs with Various f f s 





f. „ 




n 


Type I 


Type II 


Alpha." 


3onl . 


:;artlcy u 


Alpha 0 


Sor.i '.. 


Hartley 


.2S 


nr.. 

o 


o 


5 
-/ 

15 
30 


.2 560 

.2795 
.2370 


• 0355 
.0415 
.0380 


.036O 
.0450 
.0435 


.6510 

.2215 
.0340 


.3780 
.5030 
.1270 


.3775 
.5020 

. 1265 


.2^ 


■ L. J 


0 


15 
30 


.2290 

.2525 
.2465 


.0340 

.0375 
.0335 


.0350 
. 04l 5 
.0505 


• 9315 
.6175 
.2415 


• 9910 

• 8395 
.5620 


.9890 

.3715 
•5310 


.40 


.25 


0 


5 

15 
30 


.2350 
.2280 

.2375 


.0360 
.0390 
.0345 


.0375 
.0500 
. 0490 


.8450 

• 51^5 
.2045 


•9755 
.7900 
.4890 


• 9710 
.7750 
.4590 


.25 


.25 


.25 


5 

15 
30 


.1.370 
.1715 

.2:60 


.0215 
.0270 
.0275 


.0225 
. 0310 
.0340 


.9860 
.8825 
.6215 


1 .0000 

.9375 
.8945 


• 9995 
.0805 
.8600 


.40 




.25 


5 
15 
30 


.1920 

.1730 
.1975 


.0230 
.0265 
.0305 


.035. 
. 038O 

.0505 


.9590 
• 7535 
.5130 


.9965 

.9400 

.7960 


.9945 
.9165 
• 7575 



d 



a-All additional effects set at zero; that is, 



^Nominal level of .05 for each hypothesis. 

c Nominal level of .05/k for each hypothesis; k is the 

number of tests in any one experiment. 

d Nominal level for each hypothesis is not known. 



